The present invention relates to devices and methods for enhanced imaging, and more particularly, in part, to multi-spectral imaging, by mosaics and arrays of sensors designed to measure a filtered version of the light reaching them, and to methodologies for calibrating the response and processing the output of such sensors and arrays.
There is a vast amount of technology that comprises the state of the art in advanced imaging and multispectral imaging. Some background may be found, for example, in a patent to some of the present inventors: U.S. Pat. No. 6,859,275, “System and Method for Encoded Spatio-Spectral Information Processing”, Fateley et al., which is incorporated herein by reference in its entirety.
Spectral imaging systems typically have dynamic components such as moving parts or adaptive parts such as tunable filters, in order to accomplish the acquisition of spectral image information. These dynamic components are typically expensive and require complex calibration and processing.
Color cameras typically measure only three channels of color information (such as red, green and blue). As such they are limited in their use as spectral imagers. Also, color cameras typically sample colors in a so-called mosaic pattern. This means that each pixel only measures one color, with the three different color measurements happening at three different pixels—typically in a repeating pattern such as but not limited to Bayer patterns. See U.S. Pat. No. 3,971,065 to Bayer, “Color Imaging Array”, which is incorporated herein by reference in its entirety.
Therefore ordinary color cameras are limited in their capacity as advanced or spectral imaging devices for at least two reasons. First because of the limited amount of spectral information that they provide (only 3 colors). Second because the full color information is not measured per-pixel but only by mosaic. In the U.S. Pat. No. 3,971,065, Bayer states that “relative image sampling rates, by color, are in effect adjusted respective of the characteristics of human visual response.” While this is of some use for images to be viewed by humans, there is a need for more general systems, for example for scientific and engineering applications wherein human visual response is not the system response of interest. Even when images are to be viewed by humans, there is a need for an improved system for at least the following reasons. Standard techniques are available to model the missing information for a mosaic sensor, and these comprise various kinds of interpolation. Such interpolations are used to “guess” or “fill in” the missing/non-measured color information. However, these guesses can be wrong and the filled in data may, for example, appear blurry as a result. In particular, at an edge or boundary, simple interpolation will yield a result that can be far from correct. Consequently there is a need for improved methods for interpolation of missing sampled data from mosaic cameras.
Beyond spectral imaging systems, advanced imaging devices in general are needed for a variety of applications. Such advanced imaging systems include but are not limited to high resolution imaging systems, wide field-of-view imaging systems, artifact correcting imaging systems, and high dynamic range imaging systems, to name a few. In both design and use, state of the art imaging systems require tradeoffs between factors including but not limited to resolution, field-of-view, contrast, image quality, color, spectral detail and cost. Consequently there is and will always be a need for improved imaging systems that allow for better image quality of these various kinds, or, put differently, that enable better choices to be made in, for example, the tradeoffs just described.
The state of the art in advanced imaging includes mosaic imaging for enhanced imaging including multispectral imaging and improved resolution, field of view and other parameters. Examples of existing systems for such may be found in the paper “Generalized Mosaicing”, by Schechner and Nayar, Eighth International Conference on Computer Vision (ICCV'01), 2001, Volume 1, pp. 17-24, which is incorporated by reference. In that paper, mosaic imaging systems are taught that accomplish multispectral imaging, and other related designs for improved resolution, field of view and other parameters. Briefly these goals are accomplished by providing a mosaic of sensors with spatially varied properties, so that as a camera is scanned over a scene, multiple measurements of each scene point are obtained under different optical settings. While this approach accomplishes the stated goals, it requires scanning of a scene to produce the desired information. This scanning requires registration from one frame to the next, to accomplish the data fusion. However, there is information in even a single frame of image data, and it is desirable to have systems that extract as much of this information as possible. It is also desirable to have systems that, while capable of exploiting the extra information provided by scanning a scene multiple times, do not require such scanning.
In summary there is a need for devices, methods and systems for spectral imaging and advanced imaging with improved performance in one or more of image quality, contrast, resolution, field-of-view, color, spectral detail and cost, and for improved methods for interpolation of missing sampled data from mosaic imaging systems.